System and method registered video endoscopy and virtual endoscopy

ABSTRACT

An endo-robot can be tracked and a reconstructed 3D volume of data can be created for virtual endoscopy. The 3D position and orientation of each image captured by the endo-robot is determined. The longitudinal distance traveled inside the structure of interest is determined. The position, orientation and longitudinal distance are used to register the position of the endo-robot to a corresponding position inside the 3D volume of data. Virtual endoscopy can be used to locate areas of interest and correlate clinical findings with the images of the areas of interest.

This application claims priority to U.S. Provisional Application Ser.No. 60/564,936, filed on Apr. 23, 2004, which is herein incorporated byreference in its entirety.

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention relates to endoscopy, and more particularly to asystem and method for externally tracked video endoscopy usingendo-robots combined with virtual endoscopy.

2. Discussion of Related Art

Endoscopic examination is a medical procedure in which an imaging devicereferred to as an endoscope is inserted into a live subject for thepurpose of diagnosis and/or treatment. Examples of endoscopy are thecolonoscopy and bronchoscopy.

These procedures require that the instrument be inserted and controlledthrough direct mechanical interaction. This includes some portion of theinstrument being inside the patient while the rest of the sameinstrument is manipulated externally. By necessity of design, mostendoscopic devices are in the form of long flexible or rigid tubesattached to a control head that remains outside the body of the patient.

A consequence of this design is that there is usually some mild tosevere discomfort associated with endoscopy coupled with a risk ofdamage to internal structures if too much force is applied externally tothe endoscope.

Virtual endoscopy could in theory be used to explore the body structuresof interest and to detect the problem areas, such as nodules or polyps.However, this technique can lead to the detection of false positives ormisdiagnoses, which in turn can lead to unneeded invasive biopsies.

Therefore, a need exists for a system and method for an endo-robot.

SUMMARY OF THE INVENTION

According to an embodiment of the present disclosure, acomputer-implemented method for associating an image with an area ofinterest in a three-dimensional volume includes modeling a volume as thethree-dimensional volume, tracking an image capturing device within thevolume, storing position and orientation information of the imagecapturing device at a time when at least one image is captured, anddetermining a longitudinal distance traveled by the image capturingdevice at the time when the at least one image is captured. The methodfurther includes registering the position, orientation and longitudinaldistance of the image capturing device to a corresponding positionwithin the three-dimensional volume, determining an area of interestwithin the three-dimensional volume, and determining the at least oneimage registered to correspond to the area of interest, wherein the atleast one image is displayed and the area of interest is characterized.

Registering further comprises determining a centerline of thethree-dimensional volume, determining a correspondence between a trackof the image capturing device and the centerline of thethree-dimensional volume, and registering the at least one image to aposition within the three-dimensional volume. Registering furthercomprises determining an initial position of the video capturing deviceand manually selecting a position corresponding to the initial positionin the three-dimensional volume.

The longitudinal distance is a distance of the video capturing devicefrom the initial position.

Determining the at least one image comprises automatically displayingthe at least one image in conjunction with the display of thecorresponding position within the three-dimensional volume.

According to an embodiment of the present disclosure, a program storagedevice is provided readable by machine, tangibly embodying a program ofinstructions executable by the machine to perform method steps forassociating an image with an area of interest in a three-dimensionalvolume. The method steps including modeling a volume as thethree-dimensional volume, tracking an image capturing device within thevolume, storing position and orientation information of the imagecapturing device at a time when at least one image is captured, anddetermining a longitudinal distance traveled by the image capturingdevice at the time when the at least one image is captured. The methodfurther includes registering the position, orientation and longitudinaldistance of the image capturing device to a corresponding positionwithin the three-dimensional volume, determining an area of interestwithin the three-dimensional volume, and determining the at least oneimage registered to correspond to the area of interest, wherein the atleast one image is displayed and the area of interest is characterized.

According to an embodiment of the present disclosure, a method forviewing a virtual volume of a real space comprises displaying a portionof the virtual volume of the real space, displaying, automatically andsimultaneously, a registered image of the real space corresponding tothe portion of the virtual volume in a second window, and characterizingthe portion of the virtual volume based on the combination of thevirtual volume and the registered image.

BRIEF DESCRIPTION OF THE DRAWINGS

Preferred embodiments of the present invention will be described belowin more detail, with reference to the accompanying drawings:

FIG. 1 is a flow chart of a method according to an embodiment of thepresent disclosure;

FIG. 2 is a diagram of an endo-robot according to an embodiment of thepresent disclosure;

FIG. 3A is an illustration of a portion of a virtual endoscopy volumeaccording to an embodiment of the present disclosure;

FIG. 3B is an illustration of a gastrointestinal tract and associatedimages according to an embodiment of the present disclosure;

FIG. 4 is a diagram of a computer system according to an embodiment ofthe present disclosure.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

According to an embodiment of the present disclosure, images captured byan endo-robot can be registered to a reconstructed three-dimensional(3D) volume of data. The virtual volume is created based on a scan of aspace 101, e.g., a colon (see FIG. 1). The endo-robot is tracked withinthe space and 3D position and orientation of every image captured by theendo-robot is determined 102. The longitudinal distance traveled insidethe structure of interest, such as a colon, is determined 103. Theposition, orientation and longitudinal distance are used to register theposition of the endo-robot to its corresponding position inside the 3Dvolume of data 104. Virtual endoscopy can be used to locate areas ofinterest and correlate clinical findings with the images of the areas ofinterest 105. The combined clinical findings can be used to improvediagnosis confidence, decrease the number of false positives, anddecrease the number of biopsies performed. Such findings can also beused in various applications, for example, in the identification ofpatients with premalignant dysplasia in the colorectum, for example,using fluorescent imaging, enabling the identification of earlyneplasitc lesions in the colorectum, for example, usinghigh-magnification endoscopy.

Referring to FIG. 2, an endo-robot is a capsule 201 that can beswallowed or inserted non-surgically into a patient. Examples of suchrobots are those offered on the marked by such companies as GivenImaging. These Capsules include a camera 202, an illumination source 203and hardware for the wireless transmission of images 204 to a consoleoutside the patient. The camera 202 acquires images or video images,e.g., macroscopic, normal-spectrum visualization, or which can performfluorescent imaging, or magnification image or video capture.

The position and orientation of the endo-robot can be determined usingan external tracking system. The tracking is performed continuously andin real-time. The system can include a magnetic system for tracking theposition and orientation of the endo-robot during a patient exam.Various types of magnetic tracking may be used. For example, a trackingsystem can include one or more permanent magnets 205 disposed atpredetermined, known, locations within the endo-robot. Magnet locatingand tracking includes determining a dipole magnet in 3D space. An arrayof magnetic sensors disposed outside the patient collects data, e.g.,field strength of the magnet 205 at a known location, to determine boththe location and orientation of the magnet 205 and thus the endo-robotcapsule 201 including the camera 202. The array of magnetic fieldsensors can be magneto-resistive sensors arranged in groups to form setsof 3D-vector sensors. The sensors collect direction and magnitude dataof a magnetic field (e.g., described as a vector) of the magnet 205. Thevector changes direction and magnitude in a known manner in the spacesurrounding the magnet, e.g., magnetic field vectors pointperpendicularly out from the face of the magnet's north pole, and intothe magnet's south pole, and make intermediate angles to the magnet'saxis between the poles.

Another example of a tracking system includes an endo-robot having a tipelectrode, a ring electrode and a location sensor. Location isdetermined determination using three coils placed outside the patient atknown locations in three-dimensional space generating magnetic fieldsthat decay as a function of distance. The location sensor measures thestrength of a combined field, which enables a distance from each coil tobe measured. The location of the sensor and thus the endo-robot isdetermined from the intersection of the fields whose radii are thedistances measured by the sensor to the coils.

A power source 206 of the endo-robot can include a battery or inductorand battery. External powering can be realized through inductivecharging using the inductor. For inductive charging the power source 206is a secondary winding receiving charging power from a primary windingof an inductive charger. Inside the endo-robot, power is received with acoil and rectified by a diode. This direct-current (DC) voltage is usedin charging a battery. External wireless supply of energy also allowsfor longer and more power-consuming procedures to be performed by theendo-robot.

The virtual endoscopy simulates an endoscopic intervention using methodsof virtual reality and computer graphics. 3D volume data from CAT-scans,MRIs, 3D ultrasounds, rotations angiography or other sources can be usedto generate a 3D view of the inside of the respective structures. 3Dimages may be created from two-dimensional (2D) computerized tomography(CT) or magnetic resonance (MR) data, for example, by volume rendering.

Registration may be performed using a predetermined reference frame orone or more user defined correspondences between captured images and the3D volume that can form the basis for determining furthercorrespondences. An example of a correspondence can be a landmarkidentifiable in the 3D volume and captured images such as theduodenojejunal flexure.

The position, orientation and longitudinal distance associated with eachcaptured image are used to register each image to a correspondingposition inside the 3D volume of data. FIG. 3A illustrates a positionand orientation of the endo-robot as an XYZ axis within a virtualvolume, wherein, for example, the Z-axis represents a sight line of theendo-robot camera 202. The longitudinal distance is determined, e.g., bytracking position over time, and associated with images 304 asillustrated in FIG. 3B. Thus, for portions of the 3D volume that may bepositionally close, e.g., portions of the small intestine that fold ontop of one another, the longitudinal distance can be used to distinguishbetween these portions, where a first portion 302 may be 0.5 meters froman initial point 303 and a second portion may be 2.5 meters from theinitial point 303. An appropriate image, e.g., 305 is determined anddisplayed corresponding to an area of interest 302.

Registration of images to the virtual volume can use a known startingposition of the endo-robot and a position determined in the virtualvolume. For example, tracking may begin from a patient's mouth, which isan identifiable location in the virtual volume. Referring to FIG. 4,registration includes determining an initial position of the endo-robot401 and determining a position in the virtual volume corresponding tothe initial position of the endo-robot 402. The endo-robot is trackedthrough a space 403, which is represented by the virtual volume. Acorrespondence of the track of the endo-robot is determined to thevirtual volume in 3D space 404. Images captured by the endo-robot,having known positions and orientations along the track of theendo-robot are registered to corresponding positions within the virtualvolume 405. The correspondence of the track of tracked course of theendo-robot (as depicted in FIG. 3B) to the virtual volume can use acenterline of a virtual volume determined from a scan of the patient.The centerline is automatically determined as part of commerciallyavailable CT and MRI scans as the track followed by a virtualfly-through of the virtual volume. The tracked course of the endo-robotand the determined centerline will correspond in three-dimensional spaceand may be registered manually or automatically based on comparativegeometries.

The images captured by the endo-robot are registered to the volumerenderings of the virtual volume. Because each image captured by theendo-robot is associated with spatial, directional and longitudinalinformation within the volume rendering, detailed images may be viewedof areas of interest identified in the volume rendering. Registeredimages can be automatically displayed as part of a fly-through of thevirtual volume, for example, in a split screen configuration. Theregistered images can stored in memory, and can be accessed by a virtualrendering application executing by a computer, such that a virtualvolume and real image may be displayed at the same time, for example, intwo windows of an application or on two displays. Thus, a diagnosis,characterization and/or evaluation of a patient can be based on avirtual volume and a detailed registered image.

It is to be understood that the present invention may be implemented invarious forms of hardware, software, firmware, special purposeprocessors, or a combination thereof. In one embodiment, the presentinvention may be implemented in software as an application programtangibly embodied on a program storage device. The application programmay be uploaded to, and executed by, a machine comprising any suitablearchitecture.

Referring to FIG. 5, according to an embodiment of the presentinvention, a computer system 501 for implementing the present inventioncan comprise, inter alia, a central processing unit (CPU) 502, a memory503 and an input/output (I/O) interface 504. The computer system 501 isgenerally coupled through the I/O interface 504 to a display 505 andvarious input devices 506 such as a mouse and keyboard. The display 505can display views of the virtual volume and registered images. Thesupport circuits can include circuits such as cache, power supplies,clock circuits, and a communications bus. The memory 503 can includerandom access memory (RAM), read only memory (ROM), disk drive, tapedrive, etc., or a combination thereof. The present invention can beimplemented as a routine 507 that is stored in memory 503 and executedby the CPU 502 to process the signal from the signal source 508. Assuch, the computer system 501 is a general purpose computer system thatbecomes a specific purpose computer system when executing the routine507 of the present invention.

The computer platform 501 also includes an operating system and microinstruction code. The various processes and functions described hereinmay either be part of the micro instruction code or part of theapplication program (or a combination thereof) which is executed via theoperating system. In addition, various other peripheral devices may beconnected to the computer platform such as an additional data storagedevice and a printing device.

It is to be further understood that, because some of the constituentsystem components and method steps depicted in the accompanying figuresmay be implemented in software, the actual connections between thesystem components (or the process steps) may differ depending upon themanner in which the present invention is programmed. Given the teachingsof the present invention provided herein, one of ordinary skill in therelated art will be able to contemplate these and similarimplementations or configurations of the present invention.

Having described embodiments for a system and method for virtualendoscopy using registered images captured by an endo-robot, it is notedthat modifications and variations can be made by persons skilled in theart in light of the above teachings. It is therefore to be understoodthat changes may be made in the particular embodiments of the inventiondisclosed which are within the scope and spirit of the invention asdefined by the appended claims. Having thus described the invention withthe details and particularity required by the patent laws, what isclaimed and desired protected by Letters Patent is set forth in theappended claims.

1. A computer-implemented method for associating an image with an areaof interest in a three-dimensional volume comprising: modeling a volumeas the three-dimensional volume; tracking an image capturing devicewithin the volume, storing position and orientation information of theimage capturing device at a time when at least one image is captured;determining a longitudinal distance traveled by the image capturingdevice at the time when the at least one image is captured; registeringthe position, orientation and longitudinal distance of the imagecapturing device to a corresponding position within thethree-dimensional volume; determining an area of interest within thethree-dimensional volume; and determining the at least one imageregistered to correspond to the area of interest, wherein the at leastone image is displayed and the area of interest is characterized.
 2. Thecomputer-implemented method of claim 1, wherein registering furthercomprises: determining a centerline of the three-dimensional volume;determining a correspondence between a track of the image capturingdevice and the centerline of the three-dimensional volume; andregistering the at least one image to a position within thethree-dimensional volume.
 3. The computer-implemented method of claim 1,wherein registering further comprises determining an initial position ofthe video capturing device and manually selecting a positioncorresponding to the initial position in the three-dimensional volume.4. The computer-implemented method of claim 3, wherein the longitudinaldistance is a distance of the video capturing device from the initialposition.
 5. The computer-implemented method of claim 1, whereindetermining the at least one image comprises automatically displayingthe at least one image in conjunction with the display of thecorresponding position within the three-dimensional volume.
 6. A programstorage device readable by machine, tangibly embodying a program ofinstructions executable by the machine to perform method steps forassociating an image with an area of interest in a three-dimensionalvolume, the method steps comprising: modeling a volume as thethree-dimensional volume; tracking an image capturing device within thevolume, storing position and orientation information of the imagecapturing device at a time when at least one image is captured;determining a longitudinal distance traveled by the image capturingdevice at the time when the at least one image is captured; registeringthe position, orientation and longitudinal distance of the imagecapturing device to a corresponding position within thethree-dimensional volume; determining an area of interest within thethree-dimensional volume; and determining the at least one imageregistered to correspond to the area of interest, wherein the at leastone image is displayed and the area of interest is characterized.
 7. Themethod of claim 6, wherein registering further comprises: determining acenterline of the three-dimensional volume; determining a correspondencebetween a track of the image capturing device and the centerline of thethree-dimensional volume; and registering the at least one image to aposition within the three-dimensional volume.
 8. The method of claim 6,wherein registering further comprises determining an initial position ofthe video capturing device and manually selecting a positioncorresponding to the initial position in the three-dimensional volume.9. The method of claim 8, wherein the longitudinal distance is adistance of the video capturing device from the initial position. 10.The method of claim 6, wherein determining the at least one imagecomprises automatically displaying the at least one image in conjunctionwith the display of the corresponding position within thethree-dimensional volume.
 11. A method for viewing a virtual volume of areal space comprising: displaying a portion of the virtual volume of thereal space; displaying, automatically and simultaneously, a registeredimage of the real space corresponding to the portion of the virtualvolume in a second window; and characterizing the portion of the virtualvolume based on the combination of the virtual volume and the registeredimage.